Adds option to download caches using AzCopy #493
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 tasks
aiqiaoy
reviewed
Jun 5, 2020
| @@ -113,3 +113,28 @@ export async function isGnuTarInstalled(): Promise<boolean> { | |||
| const versionOutput = await getVersion('tar') | |||
| return versionOutput.toLowerCase().includes('gnu tar') | |||
| } | |||
|
|
|||
| export async function getAzCopyCommand(): Promise<string | undefined> { | |||
| // Always prefer the azcopy10 alias first, which is the correct version on Ubuntu. | |||
There was a problem hiding this comment.
What do you mean by correct version? What happens if we just do the version 10 or newer check?
There was a problem hiding this comment.
azcopy on Ubuntu is version 7.x.x, which uses a completely different CLI. So we need to use the azcopy10 alias on Ubuntu to get to version 10.
This is different on a Mac, where azcopy is v10. Windows currently does not have AzCopy installed, but I already filed an issue to fix that.
I'm not sure why it's setup this way, but this logic is needed to ensure we get the correct version on Ubuntu.
aiqiaoy
approved these changes
Jun 5, 2020
|
Closing, PR to use the NodeJS library for Azure storage used instead of AzCopy. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds option to download caches using AzCopy. This improves download speed and helps reduce flakiness. AzCopy downloads the file in 4 MB chunks, so it has more granular control over retries and parallelism.
This download option is selected when
DISABLE_AZCOPYenv var is not set (used to test the http-client download path)azcopyis available on the runnerFor hosted runners,
azcopyis available on Ubuntu and Mac, but not Windows. On Ubuntu, we use theazcopy10alias sinceazcopyrefers to a much earlier version (7.x.x).Stats
The table below shows the average time to download a 1 GB file from an EastUS storage account to a VM hosted in EastUS2 or WestUS:
TODO